Actor-Critic Reinforcement Learning for Control with Stability Guarantee

Reinforcement Learning (RL) and its integration with deep learning have achieved impressive performance in various robotic control tasks, ranging from motion planning and navigation to end-to-end visual manipulation. However, stability is not guaranteed in model-free RL by solely using data. From a control-theoretic perspective, stability is the most important property for any control system, since it is closely related to safety, robustness, and reliability of robotic systems. In this paper, we propose an actor-critic RL framework for control which can guarantee closed-loop stability by employing the classic Lyapunov's method in control theory. First of all, a data-based stability theorem is proposed for stochastic nonlinear systems modeled by Markov decision process. Then we show that the stability condition could be exploited as the critic in the actor-critic RL to learn a controller/policy. At last, the effectiveness of our approach is evaluated on several well-known 3-dimensional robot control tasks and a synthetic biology gene network tracking task in three different popular physics simulation platforms. As an empirical evaluation on the advantage of stability, we show that the learned policies can enable the systems to recover to the equilibrium or way-points when interfered by uncertainties such as system parametric variations and external disturbances to a certain extent.Comment: IEEE RA-L + IROS 202

Complexity Theory and Language Development: In celebration of Diane Larsen-Freeman

This book presents a collective effort of scholars in applied linguistics to expand the horizon of the application of Complexity Theory (CT) in the research of second language development (SLD). Ever since the inception of this paradigmatic shift initiated by Diane Larsen-Freeman in 1997, SLD research has gradually transformed to acknowledge, value, and focus on the investigation of the variability and nonlinearity of the SLD process. The curation of this volume includes first a centerpiece by Larsen-Freeman that lays out the theoretical background of CT in SLD research and nine other chapters, both theoretical and empirical, that center around the discussion of SLD as a complex dynamic system. Topics covered include the use of terminology under the CT framework, complex brain structures and language development, research on individual differences, methodological techniques and norms under CT, CT and L2 pedagogy, the multilingual systems, and the ultimate outcome of SLD

The dependence of the structure of planet-opened gaps in protoplanetary disks on radiative cooling

Planets can excite density waves and open annular gas gaps in protoplanetary disks. The depth of gaps is influenced by the evolving angular momentum carried by density waves. While the impact of radiative cooling on the evolution of density waves has been studied, a quantitative correlation to connect gap depth with the cooling timescale is lacking. To address this gap in knowledge, we employ the grid-based code Athena++ to simulate disk-planet interactions, treating cooling as a thermal relaxation process. We establish quantitative dependences of steady-state gap depth (Eq. 36) and width (Eq. 41) on planetary mass, Shakura-Sunyaev viscosity, disk scale height, and thermal relaxation timescale $(\beta)$. We confirm previous results that gap opening is the weakest when thermal relaxation timescale is comparable to local dynamical timescale. Significant variations in gap depth, up to an order of magnitude, are found with different $\beta$. In terms of width, a gap is at its narrowest around $\beta=1$, approximately $10\%$ to $20\%$ narrower compared to the isothermal case. When $\beta\sim100$, it can be $\sim20\%$ wider, and higher viscosity enhances this effect. We derive possible masses of the gas gap-opening planets in AS 209, HD 163296, MWC 480, and HL Tau, accounting for the uncertainties in local thermal relaxation timescale.Comment: 19 pages, 16 figures, 4 tables, accepted for publication in Ap

Expressing metaphorically, writing creatively: Metaphor identification for creativity assessment

Metaphor, which can implicitly express profound meanings and emotions, is a unique writing technique frequently used in human language. In writing, meaningful metaphorical expressions can enhance the literariness and creativity of texts. Therefore, the usage of metaphor is a significant impact factor when assessing the creativity and literariness of writing. However, little to no automatic writing assessment system considers metaphorical expressions when giving the score of creativity. For improving the accuracy of automatic writing assessment, this paper proposes a novel creativity assessment model that imports a token-level metaphor identification method to extract metaphors as the indicators for creativity scoring. The experimental results show that our model can accurately assess the creativity of different texts with precise metaphor identification. To the best of our knowledge, we are the first to apply automatic metaphor identification to assess writing creativity. Moreover, identifying features (e.g., metaphors) that influence writing creativity using computational approaches can offer fair and reliable assessment methods for educational settings